Distributed Data Framework Documentation

DDF relies on the Directory Permissions of the host platform to protect the integrity of the DDF during operation. System administrators MUST perform the following steps prior to deploying bundles added to the DDF.

DDF comes with a default keystore that contains certificates. This allows the distribution to be unzipped and run immediately.

If the installer was used to install the DDF and a hostname other than "localhost" was given, the user will be prompted to upload new trust/key stores.

If the hostname is localhost or, if the hostname was changed after installation, the default certificates will not allow access to the DDF instance from another machine over HTTPS (now the default for many services). The Demo Certificate Authority will need to be replaced with certificates that use the fully-qualified hostname of the server running the DDF instance.

DDF comes with a populated truststore containing entries for many public certificate authorities, such as Go Daddy and Verisign. It also includes an entry for the DDF Demo Root CA. This entry is a self-signed certificate used for testing. It enables DDF to run immediately after unzipping the distribution. The keys and certificates for the DDF Demo Root CA are included as part of the DDF distribution. This entry must be removed from the truststore before DDF can operate securely.

To create a private key and certificate signed by the Demo Certificate Authority, use the provided scripts. To use the scripts, run them out of the <INSTALL_HOME>/etc/certs directory.

The CertNew scripts:

To install a certificate signed by a different Certificate Authority, see Managing Keystores.

Finally, restart the DDF instance. Browse the Admin Console at https://<FQDN>:8993/admin to test changes.

Create a CRL in which the token issuer’s certificate is valid. The example uses OpenSSL.

$> openssl ca -gencrl -out crl-tokenissuer-valid.pem

Adding this property will also enable CRL revocation for any context policy implementing PKI authentication. For example, adding an authentication policy in the Web Context Policy Manager of /search=SAML|PKI will disable basic authentication, require a certificate for the search UI, and allow a SAML SSO session to be created. If a certificate is not in the CRL, it will be allowed through, otherwise it will get a 401 error. If no certificate is provided, the guest handler will grant guest access.

This also enables CRL revocation for the STS endpoint. The STS CRL Interceptor monitors the same encryption.properties file and operates in an identical manner to the PKI Authenication’s CRL handler. Enabling the CRL via the encryption.properties file will also enable it for the STS, and also requires a restart.

The PKIHandler implements CRL revocation, so any web context that is configured to use PKI authentication will also use CRL revocation if revocation is enabled.

With guest access, a user with a revoked certificate will be given a 401 error, but users without a certificate will be able to access the web context as the guest user.

The STS CRL interceptor does not need a web context specified. The CRL interceptor for the STS will become active after specifying the CRL file path, or the URL for the CRL, in the encryption.properties file and restarting DDF.

This guide assumes that the endpoint being created uses CXF and is being started via Blueprint from inside the OSGi container. If other tools are being used the configuration may differ.

Add the following property to the jasws endpoint in the endpoint’s blueprint.xml:

A Warning similar to the following will be displayed in the logs of the source and endpoint showing the exception encountered during certificate validation:

Limit access to the Admin Console to those users who need access. To set access restrictions on the Admin Console, consult the organization’s security architecture to identify specific realms, authentication methods, and roles required.

Limit access to the individual applications, features, or services to those users who need access. Organizational requirements should dictate which applications are restricted and the extent to which they are restricted.

If a permission is specified, any user without the required attributes will be unable to see or modify the feature, app, or configuration.

A claim is an additional piece of data about a principal that can be included in a token along with basic token data. A claims manager provides hooks for a developer to plug in claims handlers to ensure that the STS includes the specified claims in the issued token.

Claims handlers convert incoming user credentials into a set of attribute claims that will be populated in the SAML assertion. For example, the LDAPClaimsHandler takes in the user’s credentials and retrieves the user’s attributes from a backend LDAP server. These attributes are then mapped and added to the SAML assertion being created. Integrators and developers can add more claims handlers that can handle other types of external services that store user attributes.

Unauthenticated access to a secured DDF system is provided by the Guest user. Guest authentication must be enabled to allow guest users. Once the guest user is configured, redaction and filtering of metadata is done for the guest user the same way it is done for normal users.

To enable guest authentication for a context, change the Authentication Type for that context to Guest.

If a legacy client requires the use of the secured SOAP endpoints, the guest interceptor should be configured. Otherwise, the guest interceptor and public endpoints should be uninstalled for a hardened system.

A guest user’s attributes define the most permissive set of claims for an unauthenticated user.

A guest user’s claim attributes are stored in configuration, not in the LDAP as normal authenticated users' attributes are.

The karaf realm is the only realm available by default and it authenticates against the users.properties file. As JAAS authentication realms are added to the STS, more realms become available to authenticate against.

For example, installing the security-sts-ldaplogin feature adds an ldap realm. Contexts can then be pointed to the ldap realm for authentication and STS will be instructed to authenticate them against ldap.

As you add REST endpoints, you may need to add different types of authentication through the Web Context Policy Manager.

Any web context that allows or requires specific authentication types should be added here with the following format:

The fields for required attributes allows configuring certain contexts to only be accessible to users with pre-defined attributes. For example, the default required attribute for the /admin context is role=system-admin, limiting access to the Admin Console to system administrators

White listed contexts are trusted contexts which will bypass security. Any sub-contexts of a white listed context will be white listed as well, unless they are specifically assigned a policy.

Be sure to limit the hosts that are allowed to connect to the STS:

Format:

<INSTALL-DIR>/data/product-cache/<source-id>-<metacard-id>

Example:

<INSTALL-DIR>/data/product-cache/abc123

See Metacard Ingest Network Plugin.

IdP and SP are used for SSO authentication purposes.

The IdP bundles are not installed by default. They can be started by installing the security-idp feature.

The IdP client that interacts with the specified Identity Provider.

An internal Identity Provider solution.

Set the authentication types that will be accepted by the IdP.

Other authentication types can also be used in conjunction with the IDP type. For example, if you wanted to secure the entire system with the IDP, but still allow legacy clients that don’t understand the SAML ECP specification to connect, you could set /=IDP|PKI. With that configuration, any clients that failed to connect using either the SAML 2.0 Web SSO Profile or the SAML ECP spec would fall back to 2-way TLS for authentication.

The internal Identity Provider solution should be used in favor of any external solutions until the IdP Service Provider fully satisfies the SAML 2.0 Web SSO profile.

Each metacard result can contain security attributes that are pulled from the metadata record after being processed by a PolicyPlugin that populates this attribute. The security attribute is a Map containing a set of keys that map to lists of values. The metacard is then processed by a filter plugin that creates a KeyValueCollectionPermission from the metacard’s security attribute. This permission is then checked against the user subject to determine if the subject has the correct claims to view that metacard. The decision to filter the metacard eventually relies on the installed Policy Decision Point (PDP). The PDP that is being used returns a decision, and the metacard will either be filtered or allowed to pass through.

How a metacard gets filtered is left up to any number of FilterStrategy implementations that might be installed. Each FilterStrategy will return a result to the filter plugin that says whether or not it was able to process the metacard, along with the metacard or response itself. This allows a metacard or entire response to be partially filtered to allow some data to pass back to the requester. This could also include filtering any products sent back to a requester.

The security attributes populated on the metacard are completely dependent on the type of the metacard. Each type of metacard must have its own PolicyPlugin that reads the metadata being returned and then returns the appropriate attributes.

In the above example, the user’s claims are represented very simply and are similar to how they would actually appear in a SAML 2 assertion. Each of these user (or subject) claims will be converted to a KeyValuePermission object. These permission objects will be implied against the permission object generated from the metacard record. In this particular case, the metacard might be allowed if the policy is configured appropriately because all of the permissions line up correctly.

There are two options for processing filtering policies: internally, or through the use of a policy formatted in eXtensible Access Control Markup Language (XACML). The procedure for setting up a policy differs depending on whether that policy is to be used internally or by the external XACML processing engine. Setting up an internal policy is as follows:

With the security-pdp-authz feature configured in this way, the above Metacard would be displayed to the user. Note that this particular configuration would not require any XACML rules to be present. All of the attributes can be matched internally and there is no reason to call out to the external XACML processing engine. For more complex decisions, it might be necessary to write a XACML policy to handle certain attributes.

To set up a XACML policy, place the desired XACML policy in the <distribution root>/etc/pdp/policies directory and update the included access-policy.xml to include the new policy. This is the directory in which the PDP will look for XACML policies every 60 seconds. A sample XACML policy is located at the end of this page. Information on specific bundle configurations and names can be found on the Security PDP application page.

Several Policy Plugins for catalog filtering exist currently: Metacard Attribute Security Policy Plugin and XML Attribute Security Policy Plugin. These Policy Plugin implementations allow an administrator to easily add filtering capabilities to some standard Metacard types for all Catalog operations. These plugins will place policy information on the Metacard itself that allows the Filter Plugin to restrict unauthorized users from viewing content they are not allowed to view.

This document assumes familiarity with the XACML schema and does not go into detail on the XACML language. When creating a policy, a target is used to indicate that a certain action should be run only for one type of request. Targets can be used on both the main policy element and any individual rules. Targets are geared toward the actions that are set in the request. These actions generally consist of the standard CRUD operations (create, read, update, delete) or a SOAPAction if the request is coming through a SOAP endpoint.

In the examples below, the policy has specified targets for the above type of calls. For the Filtering code, the target was set for "filter", and the Service validation code targets were geared toward two services: query and LocalSiteName. In a production environment, these actions for service authorization will generally be full URNs that are described within the SOAP WSDL.

Attributes for the XACML request are populated with the information in the calling subject and the resource being checked.

The attributes for the subject are obtained from the SAML claims and populated within the XACML policy as individual attributes under the urn:oasis:names:tc:xacml:1.0:subject-category:access-subject category. The name of the claim is used for the AttributeId value. Examples of the items being populated are available at the end of this page.

The attributes for resources are obtained through the permissions process. When checking permissions, the XACML processing engine retrieves a list of permissions that should be checked against the subject. These permissions are populated outside of the engine and should be populated with the attributes that should be asserted against the subject. When the permissions are of a key-value type, the key being used is populated as the AttributeId value under the urn:oasis:names:tc:xacml:3.0:attribute-category:resource category.

To use a XACML policy, copy the XACML policy into the <DDF_HOME>/etc/pdp/policies directory.

In the event the system is unable to write to the security.log file, DDF must be configured to fall back to report the error in the application log:

The Landing Page is installed by default with a standard installation.

Configure the Landing Page from the Admin Console:

The Solr Admin user interface uses basic authentication as part of the server configuration.

The Solr server is protected by the built in REST security architecture. The configuration can be changed by editing the Web Context Policy Manager configuration for the /solr web context.

By default, the configuration is set to /solr=SAML|PKI|BASIC. This allows a user or another system to connect to Solr using any of those authentication methods.

While it is possible to encrypt the Solr index, it decreases performance significantly. An encrypted Solr index also can only perform exact match queries, not relative or contextual queries. As this drastically reduces the usefulness of the index, this configuration is not recommended. The recommended approach is to encrypt the entire drive through the Operating System of the server on which the index is located.

The basic syntax of the .config configuration files is similar to the older .cfg files but introduces support for lists and types other than simple strings. The type associated with a property must match the type attribute used in the corresponding metatype.xml file when applicable.

The following table shows the format to use for each property type supported.

It may become necessary to change the data directory after DDF has ingested data.

If export is unsuccessful, use this list to verify the correct configuration.

Because DDF is built on Apache Karaf, more information on operating DDF can be found in the Karaf documentation.

The Command Line Console is the console that is available to the user when the distribution is started manually. It may also be accessed by using the bin/client.bat or bin/client.sh scripts.

To get a list of commands, type in the namespace of the desired extension then press the Tab key.

For example, type catalog, then press Tab.

For details on any command, type help then the command. For example, help search (see results of this command in the example below).

The help command provides a description of the provided command, along with the syntax in how to use it, arguments it accepts, and available options.

The catalog:dump command was extended in DDF version 2.5.0 to provide selective export of metacards based on date ranges. The --created-after and --created-before options allow filtering on the date and time that the metacard was created, while --modified-after and --modified-before options allow filtering on the date and time that the metacard was last modified (which is the created date if no other modifications were made). These date ranges are exclusive (i.e., if the date and time match exactly, the metacard will not be included). The date filtering options (--created-after, --created-before, --modified-after, and --modified-before) can be used in any combination, with the export result including only metacards that match all of the provided conditions.

If no date filtering options are provided, created and modified dates are ignored, so that all metacards match.

Supported dates are taken from the common subset of ISO8601, matching the datetime from the following syntax:

Application commands are used from the DDF Admin application to manage applications in the DDF.

Note that no arguments are required for the subscriptions:list command. If no argument is provided, all subscriptions will be listed. A count of the subscriptions found matching the list command’s search phrase (or LDAP filter) is displayed first followed by each subscription’s ID.

The example below illustrates searching for any subscription that has "json" or "v20" anywhere in its subscription ID.

The example below illustrates searching for any subscription that has json and 172.18.14.169 in its subscription ID. This could be a handy way of finding all subscriptions for a specific site.

The arguments for the subscriptions:delete command are the same as for the list command, except that a search phrase or LDAP filter must be specified. If one of these is not specified an error will be displayed. When the delete command is executed it will display each subscription ID it is deleting. If a subscription matches the search phrase but cannot be deleted, a message in red will be displayed with the ID. After all matching subscriptions are processed, a summary line is displayed indicating how many subscriptions were deleted out of how many matching subscriptions were found.

The Command Scheduler allows administrators to schedule Command Line Shell Commands to be run in a platform-independent way. For instance, if an administrator wanted to use the Catalog commands to export all records of a Catalog to a directory, the administrator could write a cron job or a scheduled task to remote into the container and execute the command. Writing these types of scripts are specific to the administrator’s operating system and also requires extra logic for error handling if the container is up. The administrator can also create a Command Schedule, which currently requires only two fields. The Command Scheduler only runs when the container is running, so there is no need to verify if the container is up. In addition, when the container is restarted, the commands are rescheduled and executed again.

Change the timing, order, or execution of scheduled commands.

The CQL syntax used with console commands should follow the OGC CQL format. GeoServer provides a description of the grammar and examples at CQL Tutorial.

Edit the configuration file <DDF_HOME>/etc/org.ops4j.pax.logging.cfg]

The name and location of the log file can be changed with the following setting:

log4j.appender.out.file=<KARAF.DATA>/log/ddf.log

A useful way to debug and detect issues is to change the log level:

log4j.rootLogger=DEBUG, out, osgi:VmLogAppender

Set the maximum size of the log file before it is rolled over by editing the value of this setting:

log4j.appender.out.maxFileSize=20MB

Adjust the number of backup files to keep by editing the value of this setting:

log4j.appender.out.maxBackupIndex=10

By default, the DDF start scripts include a system property enabling logging of inbound and outbound SOAP messages.

-Dcom.sun.xml.ws.transport.http.HttpAdapter.dump=true

In order to see the messages in the log, one must set the logging level for org.apache.cxf.services to INFO. By default, the logging level for org.apache.cxf is set to WARN.

ddf@local>log:set INFO org.apache.cxf.services

Other appenders can be selected and configured.

For more detail on configuring the log file and what is logged to the console see: Karaf Documentation: Log.

To enable access logs for the current DDF, do the following:

Change the location of the logs to the desired location. In the settings above, location will default to data/log (same place where the log is located).

The log is using National Center for Supercomputing Association Applications (NCSA) or Common format (hence the class 'NCSARequestLog'). This is the most popular format for access logs and can be parsed by many web server analytics tools. Here is a sample output:

Monitor system logs with the LogViewer, a convenient "viewing portal" for incoming logs.

or

The LogViewer displays the most recent 500 log messages by default, but will grow to a maximum of 5000 messages. To view incoming logs, select the PAUSED button to toggle it to LIVE mode. Switching this back to PAUSED will prevent any new logs from being displayed in the LogViewer. Note that this only affects the logs displayed by the LogViewer and does not affect the underlying log.

Log events can be filtered by:

These methods are the simplest for small data sets or for testing purposes.

The DDF console application has a command-line option for ingesting files

The syntax for the ingest command is ingest -t <transformer type> <file path> relative to the installation path.

For XML data, run this command:

The Catalog application contains a Directory Monitor feature that allows files placed in a single directory to be monitored and ingested automatically. For more information about configuring a directory to be monitored, see Configuring the Content Directory Monitor.

Simply place the desired files in the monitored directory and it will be ingested automatically. If, for any reason, the files cannot be ingested, they will be moved to an automatically created sub-folder named .errors. Optionally, ingested files can be automatically moved to a sub-folder called .ingested.

Several third-party tools, such as cURL.exe and the Chrome Advanced Rest Client, can be used to send files and other types of data to DDF for ingest.

Where:
-H adds an HTTP header. In this case, Content-type header application/json;id=geojson is added to match the data being sent in the request.
-i requests that HTTP headers are displayed in the response.
-X specifies the type of HTTP operation. For this example, it is necessary to POST (ingest) data to the server.
-d specifies the data sent in the POST request. The @ character is necessary to specify that the data is a file.

The last parameter is the URL of the server that will receive the data.

This should return a response similar to the following (the actual catalog ID in the id and Location URL fields will be different):

The catalog:removeall command states you want to remove records from the local Catalog.

The --force option is used to suppress the confirmation message which asks a user if the user intentionally wants to permanently remove records from the Catalog.

The --expired option is to remove only expired records.

Metadata can be converted from one data format to another format.  Only the data format changes, but the content of the metadata does not, as long as option -p is used with the ingest command. The process for converting metadata is performed by ingesting a data file into a catalog provider in one format and dumping it out into a file in another format.